Automated detection of nitrogen deficiency in crop

ABSTRACT

Pixel color values representing an image of a portion of a field are received where each pixel color value has a respective position within the image. A processor identifies groups of the received pixel color values as possibly representing a Nitrogen-deficient plant leaf. For each group of pixel color values, the processor converts the pixel color values into feature values that describe a shape and the processor uses the feature values describing the shape to determine whether the group of pixel color values represents a Nitrogen-deficient leaf of a plant. The processor stores in memory an indication that the portion of the field is deficient in Nitrogen based on the groups of pixel color values determined to represent a respective Nitrogen-deficient leaf.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is based on and claims the benefit of U.S.provisional patent application Ser. No. 62/268,233, filed Dec. 16, 2015,the content of which is hereby incorporated by reference in itsentirety.

BACKGROUND

In order for plants to grow properly, they must have access to UV light,water and certain nutrients. For corn plants, one of the key nutrientsis nitrogen (N). Corn plants absorb mineral forms of nitrogen from thesoil. However, the amount of nitrogen available in the soil can changerapidly over time due to bacteria, water leaching, vaporization, andplant uptake. In addition, these changes can affect different parts of acorn field differently resulting in some areas have sufficient nitrogenand some areas having insufficient nitrogen. In the past, some farmshave applied extra nitrogen to the entire field to ensure that there wassufficient nitrogen for every plant. However, applying too much nitrogento a field has negative environmental consequences and increases thecosts associated with producing the crop.

SUMMARY

In one embodiment, pixel color values representing an image of a portionof a field are received where each pixel color value has a respectiveposition within the image. A processor identifies groups of the receivedpixel color values as possibly representing a Nitrogen-deficient plantleaf. For each group of pixel color values, the processor converts thepixel color values into feature values that describe a shape and theprocessor uses the feature values describing the shape to determinewhether the group of pixel color values represents a Nitrogen-deficientleaf of a plant. The processor stores in memory an indication that theportion of the field is deficient in Nitrogen based on the groups ofpixel color values determined to represent a respectiveNitrogen-deficient leaf.

In a further embodiment, a system includes an interface receiving imagedata collected by an unmanned aerial vehicle, the image datarepresenting an image of a portion of an agricultural field in thevisible spectrum. A processor processes the image data to identify aplurality of areas in the image that each possibly show aNitrogen-deficient leaf and for each identified area, identifiesfeatures of a shaped region within the area to verify that the areashows a Nitrogen-deficient leaf. The verified areas are used to store anindication of a Nitrogen level in the portion of the agricultural field.

In a still further embodiment, a computer-implemented method includeslimiting a Nitrogen-deficiency assessment of leaves in an image toselect leaves in the image by grouping pixels in the image into groupsbased on the visible color of each pixel and identifying the selectleaves as leaves that include pixels of a particular group. TheNitrogen-deficiency assessment is performed on the select leaves, wherethe Nitrogen-deficiency assessment of a leaf involves identifying askeleton for a spatial shape formed by the pixels in the particulargroup for the leaf and using the skeleton to generate shape featurevalues for the spatial shape. The shape feature values are then used toclassify the leaf into one of a plurality of Nitrogen-deficiencyclasses.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram that shows the distribution of the number ofNitrogen-deficient leaves counted versus the amount of Nitrogenfertilizer the land received before the seeding process.

FIG. 2A is a top view of a healthy leaf and FIG. 2B is a top view of aNitrogen-deficient leaf.

FIG. 3 is a RGB camera mounted on an unmanned aerial vehicle.

FIG. 4 is a flow chart that visualizes the steps followed by arecommendation scheme in accordance with one embodiment.

FIG. 5 is a graph showing collected pixel values in a three-dimensionalspace.

FIG. 6 is a graph showing the collected pixel values mapped onto a unitsphere in the three-dimensional space.

FIG. 7 is a flow chart that presents the steps that lead to the Nitrogendeficiencies assessment of an image.

FIG. 8 shows a visualization of the features extracted from theprediction of Nitrogen-deficient leaves.

FIG. 9 shows the output of the classification scheme overlaid on animage of a heavily Nitrogen-deficient portion of a field.

FIG. 10 is a block diagram of elements in an exemplary unmanned aerialvehicle.

FIG. 11 is a block diagram of a computing device the recommends segmentsand analyzes recommended segments to determine if the recommendedsegments contain Nitrogen-deficient leaves.

DETAILED DESCRIPTION

In accordance with the various embodiments, a system and method areprovided that identify which areas of a field are nitrogen-deficient sothat additional nitrogen can be applied to only those portions of thefield that need it. This system requires extensive technology because ofthe vast number of plants that are to be evaluated. For example, anaverage farm size in 2015 is 441 acres. For corn farms, more than 25,000corn plants are found in each acre of land. This means that for anaverage corn farm there are over 10 million corn plants. Assessing thesoil conditions next to each corn plant by hand is not possible. Findingways to automatically assess the soil around each corn plant is alsochallenging because of the difficulty of moving soil sensors through agrowing corn field in a fast and efficient way.

The present inventors have discovered a solution to the problem of notbeing able to test the soil next to every plant. This solution is basedon experiments by the present inventors that have shown a clearcorrelation of the number of nitrogen deficient (N-deficient) leaves ona plant with the amount of Nitrogen fertilizer that has been appliedbefore the seeding of the plants. FIG. 1 provides a graph 100 of theexperimental results showing that the density of N-deficient leaves 104decreases as the amount of applied Nitrogen per acre 102 increases.Thus, the number of N-deficient leaves is correlated to the amount ofnitrogen in the soil. The various embodiments provide methods andsystems that are able to identify and count N-deficient leaves based onimages of a corn field. The embodiments provide:

-   -   A segmentation scheme for robust and accurate clustering of        pixels based on color, and    -   A methodology for the extraction of descriptors that capture        N-deficiency in corn leaves.

The techniques used in the various embodiments improve the performanceof the computing devices used to identify areas of a field that areNitrogen deficient. In particular, the segmentation scheme greatlyreduces the amount of computing resources used to assess N-deficiency byidentifying areas of an image that are most likely to be relevant whendetermining N-deficiency. In addition, the assessment methodologyimproves accuracy of the computing device when identifying N-deficientportions of a field.

The Architecture

A. Architecture Overview

The various embodiments use a collection of visible images of a farmfield. High resolution RGB images are initially collected by asmall-scale UAV at a low altitude flight (15 m) that cover a wide areaof the field. In accordance with one embodiment, the flight issemi-automated with waypoints provided beforehand and the camera ismanually triggered once the UAV reaches a waypoint.

A visual observation reveals certain image characteristics that guidethe proposed methodology for detecting N deficiency. FIG. 2A shows ahealthy leaf 204 and FIG. 2B shows a N-deficient leaf 202. As seen inFIG. 2B, the methodology of the present embodiments capitalizes on twofeatures: (i) the unhealthy part is denoted by the yellow colored part200 of the leaf 202, and (ii) the “V” shape 206 outlined by thecomplement of the yellow component of the leaf with respect to itshealthy (green) counterpart.

The two aforementioned characteristics are directly associated with twoimplementation modules of the various embodiments. The first modulelocates rectangular regions in images that potentially includeN-deficient leaves. The second module acts as a filter of the output ofthe first module, further refining the regions so as to remove someregions such that the remaining regions are identified as containingN-deficient leaves with a high degree of confidence.

B. Hardware

The data collection process is aided by the deployment of a small-scaleUAV robot with an attached high resolution RGB camera. FIG. 3 provides aperspective view of an unmanned aerial vehicle (aerial drone) 304.Unmanned aerial vehicle 304 includes rotors such as rotors 320, 322,324, 326, 328, 330 and 332, which are controlled by a printed circuitboard in a central housing 334 and allow unmanned aerial vehicle 304 tolaunch, fly to preset locations, return to a landing area, and landautomatically. Unmanned aerial vehicle 304 includes a visible spectrumcamera 336 that can be rotated to different positions by controlmechanisms 338 under housing 334. In accordance with some embodiments,camera 336 is an RGB camera. Camera 336 can capture images with the lensof camera 336 pointing directly downward while unmanned aerial vehicle304 is at an altitude above a field.

In accordance with one embodiment, unmanned aerial vehicle has theability to carry a payload of 2500 g and when loaded with camera 336 hasa flight time of 15 minutes. In accordance with one embodiment, UAV 304includes a Wi-Fi transmitter that streams low quality video to a remotecomputer that includes a display screen. Using the display screen on theremote computer, a person can view the images being captured by camera336. A user interface shown on the display can then be used to sendcontrol messages to the Wi-Fi transmitter on UAV 306. The controlmessages are interpreted by a processor on the printed circuit board asinstructions that cause UAV 304 to change the position or orientation ofUAV 304 and/or the orientation of camera 336 to improve the images beingcaptured by camera 336.

Algorithmic Components

In this section, the two data processing modules are discussed in detailstarting with the first module (recommendation scheme) followed by thesecond module (N-deficiency assessment).

A. Recommendation Scheme

The first step towards identifying N-deficient leaves in images is torecommend image regions that hold significant information regarding thegeneral state of health of the individual leaves. This is an importantstep in the process pipeline, because it limits the computations to onlysmall image areas, thus increasing performance and reducingcomputational time.

The concept behind the recommendation algorithm is to cluster pixelsthat are of the same color together and then place bounding boxes aroundneighboring pixels that are in the same cluster. The conceptual flow ofthis recommendation module can be found in FIG. 4 in which pixels areclustered into 3 colors of pixels with the first cluster consisting ofgreen pixels, the second of yellow pixels (associated with potential Ndeficiencies), and the last of other-colored pixels corresponding to thesoil.

In accordance with one embodiment, multi-stage unsupervised clusteringis used to cluster the pixels into green, yellow and soil pixels. Beforethe clustering begins, each pixel from the RGB camera image 400 ismapped from the RGB color space 402 to another color space such as L*a*bcolor space 404 so that there are two different representations of thecolor of each pixel. Two independent clustering operations are thenperformed, one for the RGB color values and one for the L*a*b colorvalues. For each color space, the clustering operation is a two stageclustering followed by combining of the resulting clusters from eachcolor space.

In the first stage of the two-stage clustering, unsupervised clustering406, 408 is performed to form 10 clusters for each color space 402, 404,respectively. In accordance with one embodiment, K-means clustering isused with random initialization of the 10 clusters.

In the second stage, the 10 clusters of each color space are clusteredat steps 410, 412 into two respective clusters forming clusters 414 and418 for RGB color space 402 and clusters 416 and 420 for L*a*b colorspace 404. Clusters 414 and 416 contain pixels from the green clustersand clusters 418 and 420 contain pixels from all of the other clusters.In accordance with one embodiment, K-means clustering is used for thesecond stage with biased initialization in which the initializationpoint for the green clusters are selected from a normal distributionwith a mean that is deep in a green region of the color space.

The resulting two clusters 414, 418/416, 420 for each color space arethen combined into two combined color clusters 422 and 424: with cluster422 for green pixels and cluster 424 for other colored pixels. Inaccordance with one embodiment, this combination is made by forming asuperset of the pixels that are identified as belonging to the greencluster in each color space. Thus, if a pixel is in both green cluster414 and green cluster 416 it is placed in combined green cluster 422. Inaddition, if a pixel is in green cluster 414 for RGB color space 402 butis in other color cluster 420 for L*a*b color space 404, the pixel isadded to combined green cluster 422 instead of combined other colorcluster 424. Similarly, if a pixel is in green cluster 416 for L*a*bcolor space 404 but is in other color cluster 418 for RGB color space402, the pixel is added to combined green cluster 422 instead ofcombined other color cluster 424. The naive approach of segmenting thegreen parts using only a single color space is not accurate for allimages. This is especially true in images with few N-deficiencies, wherea significant variance in the representation of the green color ispresent. The accuracy of the segmentation increases when combining theclustering results of the two color spaces. This method achieves robustresults in the segmentation of green pixels for all the subject images.

In the visible spectrum, the automated distinction between yellow pixelsand pixels belonging to the soil proves to be particularly challenging.In order to successfully apply an unsupervised clustering, it is helpfulto bring the data in a form that takes advantage of the inherentproperties of the clustering algorithm employed.

FIG. 5 shows a distribution of pixel values 500 from an RGB image acrossthe three-dimensional RGB color space defined by Red axis 502, Greenaxis 504 and Blue axis 506. This distribution does not align well withthe tendency of K-means clustering to assume ellipsoid distributions fordata in each cluster. To improve the separation of yellow pixels fromsoil pixels, the pixels in cluster 424 are applied to a spacetransformation 426 to better align the data with the inherent propertiesof K-means clustering. In one embodiment, this transformation involvesprojecting the pixel values of the pixels in cluster 424 onto a unitsphere by dividing each pixel vector by the magnitude of the vector.FIG. 6 shows a distribution of transformed pixel values 600 for thepixels in yellow and soil cluster 424 across the three-dimensional RGBcolor space defined by Red axis 602, Green axis 604 and Blue Axis 606.

After the yellow and soil colored pixels have been transformed, thetransformed pixels are applied to additional K-means clustering 428 withinitialization points deep in the brown and yellow areas, resulting in arobust performance that is data driven and does not depend on humanselected thresholds. This produces soil colored pixels 430 and yellowpixels 432.

Finally, morphological operations 434 are applied to the yellow pixelsto identify spatially grouped regions of yellow pixels. Specifically,parts of the plant that are one pixel apart and are in yellow cluster432 are placed in the same spatial group. In addition, discontinuities,such as pixels of a different color that are surrounded by yellow pixelsare added to the yellow spatial group and are considered to be yellowpixels. An additional morphological step removes small groups of yellowpixels based on a threshold that considers their size. The threshold ismanually selected through a trial and error process and can fluctuatedepending on the resolution of the initial image. The smooth andsymmetrical objects that result from these morphological operationsguarantee high performance of the feature extraction step described inthe next section. The resulting spatially grouped regions of yellowpixels are spatially continuous in that each yellow pixel in thespatially grouped region is positioned next to at least one other yellowpixel in the spatially grouped region.

The surviving groups of yellow pixels are provided to a bounding boxidentifier 436 which constructs a bounding rectangle around each groupof yellow pixels. This bounding rectangle includes all of the yellowpixels in the group as well as green pixels that surround the yellowpixels and are within the bounding rectangle. This produces rectangularcandidate regions 438, which are the regions within the boundingrectangles, and excluded regions 440, which are the areas outside of thebounding rectangles. Each candidate region is then provided to thesecond module to verify that the candidate region contains a Nitrogendeficient leaf instead of a tassel or a completely yellow leaf.

B. N Deficiency Assessment

In this step, an assessment regarding the deficiency of a candidateregion is performed as shown in the flow diagram of FIG. 7. The inputsto the method of FIG. 7 consist of rectangular regions 438 suggested bythe recommendation algorithm, and depict potentially affected parts ofcorn plants captured in an image. Among the selected candidates, theleaves that are exhibiting N deficiency need to be separated from therest (e.g. tassels or stressed leaves for which assessment cannot bemade).

This distinction is based on a “V” shaped deformation that is directlyassociated with the N deficiency. To detect the “V” shaped deformation,shape or spatial feature values are identified and are applied to one ormore classifiers. For example, in accordance with one embodiment, edgedetection 444 is applied to green groups 440 in candidate region 438 andedge detection 446 is applied to yellow groups 442 in candidate region438. In accordance with one embodiment, the edge detection algorithm isbased on gradients of color and provides the edges between each yellowgroup 442 and each green group as well as the edges between one greenstructure (leaf, stem) and a neighboring green structure or soil. Thus,edge detection algorithm 444 provides edges between overlapping leaves.Once the edges have been identified, the edges for each yellow portionare applied to skeletonization algorithm 450 the edges for each greenportion next to a yellow portion are applied to skeletonizationalgorithm 448, and each skeletonization algorithm identifies a skeletonfor each portion it receives. In accordance with one embodiment, askeleton is found by sequentially removing layers of edge pixels anddesignating pixels next to the removed edge pixels as new edge pixels.During the removal process, an edge pixel is only removed if removingthe pixel will not cause a break in the group. If removing the pixelwill cause a break in the group, the pixel is maintained as an edgepixel. When no further edge pixels can be removed without causing abreak in the group, the final set of pixels represents the skeleton.

The edges detected by edge detection 446 and the skeletons identified byskeletonization algorithms 448 and 450 are applied to feature extractors452, 454, 456, and 458. In accordance with one embodiment, each featureextractor produces features that are scale and rotation invariant toaccount for the fact that the orientation of the leaves and their sizedepend on the position of the camera and the growth of the plant. Inparticular, the coordinates of the pixels belonging to skeletons andedges are normalized with respect to the size of their respectivebounding box in order to introduce scale invariance in the methodology.

In feature extractor 452, covariance matrices of the green and yellowskeletons are extracted and their eigenvalues are computed. In thiscase, the covariance matrix CϵR^(2×2) describes the distribution of thepixels of the skeleton around its centroid. The two eigenvalues σ₁, σ₂of the covariance matrix are determined and provide the dispersion ofthe pixels along the direction of the two eigenvectors of C. The ratioof the two eigenvalues σ₂/σ₁ is output for each green and yellowskeleton in candidate region 438 as the extracted features from featureextractor 452. Each of these features encapsulates the shape of arespective skeleton.

Feature extractor 456 determines histograms of minimum distances betweenthe yellow skeleton and the green skeletons. Feature extractor 456determines histograms of minimum distance between the yellow skeletonand the edges of the yellow region in candidate region 438. Thisapproach ensures the rotation invariance of the features. Particularly,the histogram of minimum distance for pixel i along the skeleton s iscomputed as follows: is calculated asd _(i) =inf∥s _(i) −e _(j) ∥,∀j=1, . . . ,n _(e) i=1, . . . ,n _(s)  (1)where n_(s) is the number of pixels of the skeleton, s_(i) are thepixels of the skeleton, e_(j) are the pixels of the edges and n_(e) isthe number of pixels of the edges. To compute the minimum distancebetween each pixel of a skeleton and another skeleton, the pixels of theedges in equation 1 are replaced with the pixels of the other skeleton.If the histogram of distances is seen as a distribution of points, thenthe features that characterize it are the first four moments of thisdistribution: (i) the mean, (ii) the variance, (iii) the skewness, and(iv) the kurtosis, which are the feature vectors output by featureextractors 456 and 458 for each distance histogram.

Feature extractor 454 scans candidate region 438 vertically andhorizontally counting how many times the skeleton of a green part isencountered in each row and column and provides the count for each rowand column of pixels as an output feature.

FIG. 8 provides a visualization of a recommended segment/candidateregion 600 showing the various features that are extracted in theprocess shown in FIG. 7. Recommended segment 600 is an image of a leaf601 with ground 602 in the background. Leaf 601 includes two greenportions 604 and 606 that are separated by a yellow portion 608. Thesolid round circles, such as solid round circle 610, show the edge ofthe green portions and the solid squares, such as solid square 612 showthe edges of yellow portion 608. The x's, such as x 614, show theskeletons 616 and 618 of the green portions and the asterisks, such asasterisk 620, show the skeleton 622 of yellow portion 608. Arrow 624shows a minimum distance between a point on skeleton 622 and the edge ofyellow portion 608 and arrow 626 shows the minimum distance between apoint on skeleton 622 and skeleton 616. Vertical arrow 630 represents avertical scan of segment 600 and shows two intersection points 632 and634, where each intersection point is an intersection with one of thegreen portion skeletons 616 and 618. Horizontal arrow 636 represents ahorizontal scan of segment 600 and shows one intersection point 638 withgreen portion skeleton 618.

As presented in FIG. 8, for a N-deficient leaf, the distribution of theskeletons tends to be elongated towards one direction, the distancesbetween the skeleton and edges of the yellow part are distributedsymmetrically around the yellow part's skeleton in an almost consistentdistance, and the skeleton of the green part that is enclosing theyellow part is encountered two times during the rows' and columns'scanning. On the other hand, non-N-deficient suggestions generally havea cyclic distribution of skeletons, the distances between skeletons andedges are irregular, and in the case of tassels the scanning processwould encounter the same skeleton multiple times.

The features extracted by feature extractors 452, 454, 456 and 458 areprovided to a Logistic Regression classifier 460 that has been trainedon features extracted from labeled image regions where some of theregions include N-deficient leaves and some regions do not. LogisticRegression is selected over Naive Bayes and SVM with linear kernels,because it achieves a better overall accuracy. As shown later, theimbalance in the number of queries between the two classes of theclassification introduces problems regarding the sensitivity of themodel for the Logistic Regression classifier. Sacrificing the accuracyfor a better balance between sensitivity and specificity can be achievedthrough an SVM classifier with RBF kernels. Logistic regressionclassifier 460 then outputs a final indication 462 of whether the regionincludes a N-deficient leaf.

The number of regions that are classified as containing an N-deficientleaf by Logistic regression classifier 460 are then stored in memorytogether with an identifier for the part of the field captured in theimage. This count can then be combined with counts from other images toidentify parts of the field that are N-deficient and therefore requirethe application of additional Nitrogen. For example, a threshold numberof leaves can be set such that when the count exceeds the threshold, thepart of the field captured in the image is designated as beingN-deficient. Thus, the count can be used to store a current Nitrogenlevel for various parts of an agricultural field.

The results in FIG. 9 show a collection of candidate regions 438 in animage where the candidate regions shown by dash-dot-dash boxes, such asregion 900, were classified as N-deficient by Logistic Regressionclassifier 460 and candidate regions shown in dashed boxes, such asregion 902, were classified as not being N-deficient by LogisticRegression classifier 460.

Experimental Results

Prior to the presentation of the results, it is important to address amajor obstacle when dealing with the visible spectrum imaging, which isthe illumination inconsistency. The application of the proposed schemeto a real world setting requires the assembly of information regardingthe weather conditions during the flight. The findings based on theNational Climatic Data Center of the National Oceanic and AtmosphericAdministration (NOAA) show that 30% of the days of the year that corn isbeing grown the sunshine provides ideal illumination for imaging, whileabout 60% of the same period of time the existing weather conditionsadvocate for an acceptable analysis of RGB images. These findingssuggest that the proposed architecture is capable of providinginformation throughout the biggest portion of the corn growing cycle.

The dataset that was processed consists of 39 high-resolution RGB imagesgathered by UAV 304 (FIG. 3) over two days and a time period between 10am and 1 pm. Twenty-one images show a severe deficiency state where no Nfertilizer was applied and eighteen images depict a state where 280lbs/acre of N were applied before the seeding process. All plants wereplanted the same day and the soil differences in nutrients and color arestatistically insignificant. The images collected by the UAV weretransferred to a remote station that handled the offline computationsupon the completion of the flight.

A. Recommendation Scheme

The first step of the proposed architecture includes the validation ofthe accuracy of the segmentation algorithms as well as the performanceof the recommendation scheme.

TABLE I Green Pixels Segmentation Accuracy in Different Color SpacesDeficiency Level RGB L*a*b Combined Spaces Heavy 96.0% 97.0% 96.4% Light70.2% 51.7% 92.3%

TABLE II Accuracy, Sensitivity, and Specificity for Logistic Regression(LR) and SVM with RBF Kernels (SVM) of Different σ Parameters SVM SVMSVM SVM SVM LR (σ = 1) (σ = 2) (σ = 4) (σ = 6) (σ = 8) Acc. 79.2% 74.4%73.7% 72.3% 70.8% 68.9% Sens. 29.5% 19.3% 55.0% 62.1% 61.8% 62.7% Spec.95.0% 94.2% 80.4% 76.0% 74.0% 71.1%

The results for the green pixels' segmentation are summarized in TableI. These results were computed with the help of 4 hand-drawn masks (2for each case of severe and light deficiency) created on 4 differentimages.

It is evident from the last column of Table I that the mixed spaceshierarchical K-means scheme performs better than a naive single colorspace K-means approach, since it remains consistently accurate for thedifferent levels of N deficiency (heavy and light).

Applying the recommendation algorithm on all the captured imagesresulted in the creation of 1279 queries including N-deficient leaves,tassels, and non-N-deficient yellow regions. Three hundred and eleven(311) of the queries were groundtruthed as N-deficient, while theremaining nine hundred and sixty eight (968) were assessed as beingnon-N-deficient. The percentage of N-deficient leaves that were missedduring the suggestion phase was manually estimated at 5.3% for theseverely deficient case and 23.1% for the images with the healthierplants. The undetected areas in the second case are due to the heavyocclusion and the absence of illumination in the lower leaves of theplants making their discernment challenging even for a human. Thisoutcome does not undermine the value of the proposed methodology, sinceit applies to plants whose yield is not significantly affected by thelack of N.

B. N Deficiency Assessment

The classification models were trained on a subset of 1279 queries andapplied to a test set to measure the performance of the method followinga 10-fold validation scheme. The classification of N-deficient leavesversus the rest non-N-deficient suggestions for a Logistic Regressionclassifier achieved 84.2% correct classification for the heavilydeficient cases and 72.9% for the light deficient case. In the secondcase, several suggestions that represent tassels were falsely assignedas N-deficient resulting in a drop of performance when compared to thefirst case. Combining both cases, an overall 79.2% accuracy was reached,with the specificity and sensitivity of the Logistic Regression modelbeing 95% and 29.5% respectively. The high specificity percentage showsthat the algorithm is particularly capable of detecting suggestions thatare truly non-N-deficient, while the sensitivity result suggests that itlacks the ability to robustly identify the true N-deficient leaves.

The three-to-one ratio between the number of queries of the two classes(968 to 311) is an important factor that relates directly to theperformance of the Logistic Regression. Essentially, this ratio favorsthe selection of more samples from the non-N-deficient queries duringthe training process, biasing the final parameter estimation of theclassifier.

As suggested earlier, it is possible to achieve a better balance betweenspecificity and sensitivity by utilizing a SVM classifier with RBFkernels. Table II presents the accuracy, sensitivity, and specificityfor the Logistic Regression versus SVM classifiers with several sigmaparameters. These results show that it is possible to attain a bettersensitivity outcome with the sacrifice of accuracy. Depending on thedesired outcome of the application, different classification models maybe used. For example, it is possible to use the Logistic Regressionapproach to robustly identify the leaves that are not N-deficient andredirect the attention of the farmer to a smaller number of leaves thatare more probable to exhibit N deficiency. On the other hand, exploitingthe SVM with RBF kernel models can achieve a balanced classificationoutcome able to successfully suggest true N-deficient leaves.

The results of the proposed methodology support the choice of usingordinary images in the visible spectrum, taken by a sensor that hassignificantly lower cost than its rivals that operate in the invisiblespectrum. A performance of 84.2% is achieved for the correctclassification between N-deficient leaves and non-N-deficient yellowimage segments. This result sets a strong basis for more elaboratedattempts towards the utilization of RGB imaging for close up precisionagriculture in fields.

FIG. 10 provides a block diagram of elements found in unmanned aerialvehicle 304 including a printed circuit board 1000, motors 1016, flightsensors 1018, and Global Positioning Satellite (GPS) module 1020.Although FIG. 10 shows a single printed circuit board 1000, thecomponents on printed circuit board 1000 may be distributed acrossmultiple printed circuit boards in unmanned aerial vehicle 304. Printedcircuit board 1000 includes a memory 1002, a processor 1004, a motorinterface 1006, a flight sensor interface 1008, GPS interface 1010,camera interface 1012 and a wireless communication module 1014. Motorinterface 1006 acts as an interface between processor 1004 and one ormore motors 1016 used to turn the rotors of unmanned aerial vehicle 304.Flight sensor interface 1008 provides an interface between processor1004 and one or more flight sensors 1018 that are able to detect theorientation, velocity and acceleration of unmanned aerial vehicle 304along three separate axes. Flight sensors 1018 can further determinepitch, roll and yaw of unmanned aerial vehicle 304. GPS interface 1010provides an interface between GPS module 1020 and processor 1004. GPSmodule 1020 can detect signals from multiple satellites to determine aposition of unmanned aerial vehicle 304. This position information caninclude a latitude and longitude of unmanned aerial vehicle 304 as wellas the altitude of unmanned aerial vehicle 304. Camera interface 1012provides an interface between processor 1004 and camera 336. Camerainterface 1012 allows processor 1004 to set parameters for capturingimages in camera 336 and to initiate the capturing of images. Camerainterface 1012 also relays captured image data from camera 336 toprocessor 1004 for storage in memory 1002.

Wireless communication subsystem 1014 can include one or morecommunication modules for communicating with other devices using one ormore communication protocols. For example, wireless communications 1014can support wireless LAN, short-range radio communications, cellulardata services and satellite communications. Wireless communicationsubsystem 1014 allows unmanned aerial vehicle 304 to communicate with aremote base station (not shown).

Memory 1002 includes data and computer-executable instructions to allowprocessor 1004 to launch unmanned aerial vehicle 304, to attain adesired altitude at a desired image capture location, to fly betweenimage capture locations (also referred to as collection locations orwaypoints), to return to a landing area and to land. In particular,memory 1002 includes flight control instructions 1050 that are used byprocessor 1004 to control motors 1016 based on sensor data from flightsensors 1018 so that unmanned aerial vehicle 304 maintains stable flightand is able to launch, land and fly between way points along a path.Memory 1002 also includes waypoints and landing locations 1052 which areused by processor 1004 to know where unmanned aerial vehicle 304 shouldfly to when it is launched, where camera 336 should be instructed tocapture images along the flight path, and where unmanned aerial vehicle304 should land. Memory 1002 also includes camera controls 1054 whichare settings for camera 336 that are to be used when capturing images.

Image data captured by camera 336 are stored by processor 1004 alongwith time and date information, camera settings information and cameraposition and orientation information as image data 1056. Thus, for eachimage captured by camera 336, processor 1004 augments the image datawith metadata that describes the time and date the image was captured,the position and orientation of camera 336 when the image was capturedand the camera settings such as filters used by the camera when theimage was captured.

An example of a computing device that can be used to identifyrecommended segments and analyze the segments to identify N-deficientleaves in the various embodiments is shown in the block diagram of FIG.11. The computing device 10 of FIG. 11 includes a processing unit 12, asystem memory 14 and a system bus 16 that couples the system memory 14to the processing unit 12. System memory 14 includes read only memory(ROM) 18 and random access memory (RAM) 20. A basic input/output system22 (BIOS), containing the basic routines that help to transferinformation between elements within the computing device 10, is storedin ROM 18. Computer-executable instructions that are to be executed byprocessing unit 12 may be stored in random access memory 20 before beingexecuted.

Embodiments of the present invention can be applied in the context ofcomputer systems other than computing device 10. Other appropriatecomputer systems include handheld devices, multi-processor systems,various consumer electronic devices, mainframe computers, and the like.Those skilled in the art will also appreciate that embodiments can alsobe applied within computer systems wherein tasks are performed by remoteprocessing devices that are linked through a communications network(e.g., communication utilizing Internet or web-based software systems).For example, program modules may be located in either local or remotememory storage devices or simultaneously in both local and remote memorystorage devices. Similarly, any storage of data associated withembodiments of the present invention may be accomplished utilizingeither local or remote storage devices, or simultaneously utilizing bothlocal and remote storage devices.

Computing device 10 further includes a hard disc drive 24, an externalmemory device 28, and an optical disc drive 30. External memory device28 can include an external disc drive or solid state memory that may beattached to computing device 10 through an interface such as UniversalSerial Bus interface 34, which is connected to system bus 16. Opticaldisc drive 30 can illustratively be utilized for reading data from (orwriting data to) optical media, such as a CD-ROM disc 32. Hard discdrive 24 and optical disc drive 30 are connected to the system bus 16 bya hard disc drive interface 32 and an optical disc drive interface 36,respectively. The drives and external memory devices and theirassociated computer-readable storage media provide nonvolatile storagemedia for the computing device 10 on which computer-executableinstructions and computer-readable data structures may be stored. Othertypes of media that are readable by a computer may also be used in theexemplary operation environment.

A number of program modules may be stored in the drives and RAM 20,including an operating system 38, one or more application programs 40,other program modules 42 and program data 44. In particular, applicationprograms 40 can include programs for executing the methods describedabove including clustering, SVM classification, morphological operators,bounding box identification, skeletonization, edge detection,identification of covariance matrices and eigenvalues of those matrices,row-column scanning, distance histogram formation and logisticregression. Program data 44 may include image data, feature data, classlabels, cluster probability functions, classifier accuracy, classifierweights, labeled data, classifier scores and class labels.

Input devices including a keyboard 63 and a mouse 65 are connected tosystem bus 16 through an Input/Output interface 46 that is coupled tosystem bus 16. Monitor 48 is connected to the system bus 16 through avideo adapter 50 and provides graphical images to users. Otherperipheral output devices (e.g., speakers or printers) could also beincluded but have not been illustrated. In accordance with someembodiments, monitor 48 comprises a touch screen that both displaysinput and provides locations on the screen where the user is contactingthe screen.

The computing device 10 may operate in a network environment utilizingconnections to one or more remote computers, such as a remote computer52. The remote computer 52 may be a server, a router, a peer device, orother common network node. Remote computer 52 may include many or all ofthe features and elements described in relation to computing device 10,although only a memory storage device 54 has been illustrated in FIG.11. The network connections depicted in FIG. 11 include a local areanetwork (LAN) 56 and a wide area network (WAN) 58. Such networkenvironments are commonplace in the art.

The computing device 10 is connected to the LAN 56 through a networkinterface 60. The computing device 10 is also connected to WAN 58 andincludes a modem 62 for establishing communications over the WAN 58. Themodem 62, which may be internal or external, is connected to the systembus 16 via the I/O interface 46.

In a networked environment, program modules depicted relative to thecomputing device 10, or portions thereof, may be stored in the remotememory storage device 54. For example, application programs may bestored utilizing memory storage device 54. In addition, data associatedwith an application program, such as data stored in the databases orlists described above, may illustratively be stored within memorystorage device 54. It will be appreciated that the network connectionsshown in FIG. 11 are exemplary and other means for establishing acommunications link between the computers, such as a wireless interfacecommunications link, may be used.

Although the present invention has been described with reference topreferred embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

What is claimed is:
 1. A method comprising: receiving pixel color valuesrepresenting an image of a portion of a field, each pixel color valuehaving a respective position within the image; a processor identifyinggroups of the received pixel color values; for each group of pixel colorvalues: the processor converting the pixel color values into featurevalues that describe a shape of the positions of the group of pixelcolor values; and the processor using the feature values describing theshape to determine whether the group of pixel color values represents anitrogen deficient leaf of a plant; and the processor storing in memoryan indication that the portion of the field is deficient in nitrogenbased on the groups of pixel color values determined to represent arespective nitrogen deficient leaf.
 2. The method of claim 1 whereinidentifying groups of the received pixel color values comprises:classifying pixel color values into one of two classes, the two classescomprising a first class containing pixel color values determined to berepresentative of a particular color and a second class containing pixelcolor values determined to not be representative of the particularcolor; identifying regions of pixel color values of the first class,where each region comprises a plurality of pixel color values in thefirst class such that each pixel color value of the first class has aposition next to a position of at least one other pixel color value ofthe first class in the image; defining positions of a respectivebounding box for each region, such that the respective bounding boxsurrounds the positions of the pixel color values of the region andidentifying the pixel color values with positions surrounded by thebounding box.
 3. The method of claim 2 wherein defining positions of arespective bounding box comprises defining the positions of therespective bounding box such that pixel color values of the second classthat have positions outside the region are within the bounding box. 4.The method of claim 3 wherein converting the pixel color values intofeature values comprises identifying a skeleton for the region.
 5. Themethod of claim 4 wherein converting the pixel color values into featurevalues further comprises identifying eigenvalues of a covariance matrixfor the skeleton of the region.
 6. The method of claim 4 whereinconverting the pixel color values into feature values comprisesidentifying edges of the region.
 7. The method of claim 6 whereinconverting the pixel color values into feature values further comprisesdetermining distance values between the skeleton for the region and theedges of the region.
 8. The method of claim 7 wherein converting thepixel color values into feature values further comprises identifying twoareas of pixel color values outside of the region and within thebounding box and identifying a skeleton for each area.
 9. The method ofclaim 8 wherein converting the pixel color values into feature valuesfurther comprises determining distances between the skeleton of theregion and the skeletons of the areas.
 10. The method of claim 8 whereinconverting the pixel color values into feature values further comprisesperforming a plurality of row scans and for each row scan counting thenumber of area skeletons that are in the row.
 11. A system comprising:an interface receiving image data collected by an unmanned aerialvehicle, the image data representing an image of a portion of anagricultural field in the visible spectrum; a processor processing theimage data to: identify a plurality of areas in the image; for eachidentified area, identifying features from the image data wherein thefeatures describe a shape of a shaped region within the area to verifythat the area shows a nitrogen deficient leaf; and using the verifiedareas to store an indication of a nitrogen level in the portion of theagricultural field.
 12. The system of claim 11, wherein identifying theplurality of areas comprises: grouping pixels based on a respectivecolor value of each pixel; for at least one group of pixels, identifyingregions were the pixels in the group are spatially continuous; and foreach region, identifying an area that includes the region as one of theplurality of areas.
 13. The system of claim 11 wherein identifyingfeatures that describe a shape of a shaped region within an areacomprises identifying a skeleton of the shaped region.
 14. The system ofclaim 13 wherein identifying features that describe a shape of a shapedregion further comprises determining a covariance matrix of theskeleton.
 15. The system of claim 13 wherein identifying an area thatincludes the region comprises identifying an area that includes twoadditional regions of spatially continuous pixels and wherein theprocessor further determines a skeleton of each of the two additionalregions.
 16. A computer-implemented method comprising: limiting anitrogen deficiency assessment of leaves in an image to select leaves inthe image by grouping pixels in the image into groups based on thevisible color of each pixel and identifying the select leaves as leavesthat include pixels of a particular group; and performing the nitrogendeficiency assessment of the select leaves, the nitrogen deficiencyassessment of a leaf comprising: identifying a skeleton for a spatialshape formed by the pixels in the particular group for the leaf; usingthe skeleton to generate shape feature values for the spatial shape; andusing the shape feature values to classify the leaf into one of aplurality of nitrogen deficiency classes.
 17. The computer-implementedmethod of claim 16 wherein grouping pixels in the image into groupsbased on the visible color of each pixel comprises grouping the pixelsinto groups using two different color spaces.
 18. Thecomputer-implemented method of claim 16 wherein using the skeleton togenerate shape feature values comprises identifying eigenvalues of acovariance matrix of the skeleton and using the ratio of the eigenvaluesas a shape feature value.
 19. The computer-implemented method of claim16 wherein using the skeleton to generate shape feature values comprisesidentifying edges of the spatial shape and using distances from theskeleton to the edges to generate the shape feature values.
 20. Thecomputer-implemented method of claim 16 wherein the nitrogen deficiencyassessment of a leaf further comprises identifying skeletons of twospatial shapes formed by pixels in a second group for the leaf and usingthe skeletons of the two spatial shapes to generate additional shapefeature values used to classify the leaf into one of the plurality ofnitrogen deficiency classes.